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ABSTRACT 



This is a compilation of the Executive Summaries from 
several documents: "Professional Development District Team Survey Results" 
(July 1997) , "CSL Assessment Training Survey Results" (January 1999) , "CSL 
Assessment Professional Development Evaluation Additional Data Analysis of 
District Surveys" (February 1999) ; "1998-99 CSL Assessment Site Visit 
Results" (June 1999) ; "1998-99 CSL Assessment District Contact Interview 

Results" (September 1999) ; and "The Washington Professional Development 
Program in Assessment Literacy, Final Report" (May 2000) . The Washington 
Assessment Professional Development Program was designed to promote the 
effective use of high-quality state and classroom assessments throughout the 
state of Washington. Since 1995, Assessment Training Centers have operated in 
intermediate service districts and other already established entities in 
Washington. The professional development model emphasized pass-through 
training of trainers and locally situated collaborative learning teams. The 
program depended on several formal and informal means of gathering 
information to evaluate its processes and impact. These included: (1) 

standard evaluation forms completed at workshops; (2) records of numbers and 
types of services; (3) ongoing informal collection of data; and (4) two 
formal evaluations of program impact in 1996-1997 and 1998-1999. In the 
1996-1997 evaluation, paper-and-pencil surveys were sent to 11 Assessment 
Center directors, 66 staff members, more than 247 assessment contacts, and 
370 principals. From responses on these surveys, 20 training contacts were 
chosen for telephone interviews in the 1998-1999 evaluation. Site visits were 
conducted during both evaluations. Results of both evaluations indicate that 
the program has a large volume of training materials that both the Center and 
school district teams found very useful. Overall, the teams were pleased with 
the services they were provided. The "pass-through" training did not work as 
well as specifying the goals for training and allowing the Center teams to 
develop their own workshops and service delivery methods. It was also found 
that educators need to see lots of sample assessments that are aligned with 
state standards, and that changes in classroom practice require support from 
administrators . Recommendations for program improvement were made based on 
the findings from both evaluations. (SLD) 
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1 Presented as part of the symposium Five Years of Classroom Assessment Development: A Washington State 
Showcase at the 2001 NCME Annual Meeting in Seattle, WA. 

2 This is a compilation of the Executive Summaries from several documents: Professional Development District 
Team Survey Results (July 1997), CSL Assessment Training Survey Results (January 1999), CSL Assessment 
Professional Development Evaluation Additional Data Analysis of District Surveys (February 1999); 1998-99 CSL 
Assessment Site Visit Results (June 1999); 1998-1999 CSL Assessment District Contact Interview Results 
(September 1999), The Washington Professional Development Program in Assessment Literacy, Final Report (May 
2000). All of these reports are available from the Office of the Superintendent of Public Instruction, Becky 
Turnbull, Old Capitol Building, PO Box 47200, Olympia 98504. 




2 



BEST COPY AVAILABLE 
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Evaluation Results 



Program Description 

Mission 

The mission of the Assessment Professional Development Program was/is to promote the 
effective use of high-quality state and classroom assessments throughout the State of 
Washington. 

The five-year program funded by the state (1995-2000) was launched to support the 
comprehensive school reform effort in Washington. The reform effort had/has several related 
aspects. One aspect was the development and dissemination of content standards. Another 
aspect involved transforming those standards into new criterion-referenced state assessments and 
helping Washington educators understand them as thoroughly as possible from the start. The 
third aspect was to help educators statewide develop the assessment literacy needed to (a) use 
state assessment materials and results wisely, and (b) develop and use classroom assessments to 
help students succeed at attaining state standards. 

Goals 

The assessment literacy professional development program addressed each of these aspects. 
Specifically, the goals of the program were to help educators 

1 . Understand the state standards — Essential Academic Learning Requirements (EALRs) — as a 
foundation of sound classroom instruction and assessment 

2. Understand and learn to apply standards of quality assessment to day-to-day classroom 
assessment development and use 

3. Develop proficiency in using student-involved classroom assessment, record keeping and 
communication as instructional interventions 

4. Understand the development of the new state assessments — the Washington Assessment of 
Student Learning (WASL) — and how to interpret and use the results. 

Organizational Structure 

Since 1995, Assessment Training Centers have operated in intermediate service districts and 
other already-established entities (e.g., the state department of education, Washington Education 
Association, and district consortia). At the time state support for the effort ended, in 2000, there 
were 14 such Centers. Centers have been self-supporting since 2000. 

The original professional development model emphasized a combination of “pass-through” 
training of trainers and locally situated collaborative learning teams. For the pass-through 
component, the state-level project team (state staff, consultants, and contractors) trained Center 
Teams, who then provided professional development for local school district assessment training 
teams (District Teams). Ultimately it was the responsibility of the local District Team to train 
local teachers and encourage the formation of learning teams for the training of its faculty. 
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To provide the first level of training, the state convened training of trainer meetings for all 
Center staff two to three times a year. The purpose of these meetings was to share new EALR 
and WASL developments, review new training materials, and provide helpful insights about 
assessment literacy development. At the beginning, the meetings placed more emphasis on 
demonstrating structured training modules to “pass-through” to District Teams. As time 
progressed, the meetings became more informational, with possible training activities still 
modeled (by both state and center staff), but presented more as ideas to throw into the training 
mix created by Centers. 

An adjunct to the pass-though modules was learning teams. (In fact, the goal of some of the 
pass-though modules was to stimulate interest in learning teams.) The majority of learning about 
classroom assessment would then take place in the context of learning teams. This was actually 
one of more innovative aspects of the program. Complete materials were, provided to local 
educators to set up and conduct learning teams on classroom assessment. 

The mechanisms used by Center Teams to reach out to districts varied greatly from team to team. 
Each Center sensed the needs of its districts and, within guidelines given by the state, tailored its 
workshop offerings and other services to those needs. The typical pattern, however, was for 
Center Teams to follow biannual training of trainer meetings with their own workshops for 
District Teams and local faculties covering the same material in roughly the same way. This 
increasingly became interspersed with other services tailored to local needs. 

Centers provided other services to client districts besides training — setting up and conducting 
collaborative learning teams to pursue assessment professional development in depth, technical 
assistance, and lending libraries. Lending libraries were established for the efficient availability 
of a broad variety of training materials, information, and expensive training adjuncts such as 
videos. 

Professional Development Content 

The content of the Assessment Professional Development Program centered on each of the key 
facets of the program. As the EALRs and WASL were developed and came on line over the 
course of the project, materials and strategies were developed and implemented to help educators 
understand state content standards, the test development process, the nature of the assessments 
being created, and the meaning of the test scores. 

In addition, the program focused on five specific standards of classroom assessment quality, 
from the Assessment Training Institute (ATI), that teachers would learn to use to maximize the 
accuracy and usefulness of their classroom assessments. Similarly, ATI procedures and 
materials for conducting professional development in classroom assessment through the use of 
building-situated collaborative learning teams were used 3 . 



3 Descriptions of the EALRs, WASL, professional development materials developed by the state to help educators 
understand WASL and the EALRs, and ATI approaches and professional development materials can be found in the 
reports cited earlier, by calling Beck Turnbull at the state department of education (360-753-6738), or by contacting 
the Assessment Training Institute (503-228-3060). 
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Program Evaluation 

Description 

The program depended on several formal and informal means of gathering information about 

processes and impact in order to formatively inform content and process decisions: 

• A standard evaluation form for each workshop — both those given to Center Teams by the 
state and those given to District Teams by the Centers. 

• Records of numbers and types of services. 

• On-going informal collection of information. For example: (1) a pilot of teacher assessment 
growth portfolios in 1998; and (2) anecdotal reports of impacts associated with the program 
such as a revision in teacher licensing standards, changes in assessment training in 
institutions of higher education, and PTA work in assessment literacy for parents. 

• Two formal evaluations (1996-87 and 1998-99) of program impact on Centers, districts, and 
individual educators. 

The last two provide most of the “hard” data reported on here and so require more description. 

These evaluations examined both the service delivery process and the impact of the assessment 

professional development effort. 

Process evaluation information collected included: 

1 . The amount and nature of professional development services the Centers provided to the 
districts in their regions. 

2. The amount and nature of professional development services the District Teams provided to 
teachers. 

3. The prevalence of the use of collaborative learning teams as a professional development 
strategy. How these learning teams were set up. 

4. The prevalence of training that centered on the Tool Kits — compendiums of assessment 
activities, aligned with state standards, that teachers could use in the classroom. 

5. How Regional Centers were set up — philosophy, goals, services provided, staffing, 
use/modification of materials received from the state, and how configuration has changed 
over time. 

Impact evaluation information collected included: 

1 . Center staff satisfaction with the training, support and resources provided to them by the state 
and its contractors. Suggestions for refining future assistance. 

2. District client satisfaction with the training, support and resources provided by the Regional 
Centers. Suggestions for refining future assistance. 

3 . Changes in the quality of teacher classroom assessments. 

4. Changes in the assessment confidence of all participants in the effort — Center directors and 
staff, district contacts, administrators that participated in training, and teachers that 
participated in training. 
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Because of limits on the funding available to conduct the two evaluations, they necessarily had to 
rely heavily on answers to survey questions. However, each evaluation did have site visit and/or 
telephone interview components. 

Paper and Pencil Surveys. For the 1996-97 evaluation, surveys were mailed to all 286 District 
Team contacts — those individuals directly served by Regional Centers. The surveys were 
intended to discover what district contacts thought of the services provided thus far and how this 
training was being “passed-through” to local teachers. 

For the 1998-99 evaluation, paper and pencil surveys were sent to: 

• All Assessment Center directors with responsibilities to provide training (N = 1 1) 

• All Assessment Center staff (N = 66). 

• A sample of district assessment contacts (N more than 247 — some surveys were sent by 
Centers who did not report the total number sent, only the number returned) 

• A sample of principals attending a series of workshops designed just for them (N = 370). 

Telephone Interviews. Telephone interviews were conducted only as part of the 1998-99 
evaluation. From responses on the 1998 surveys, 20 district training contacts were chosen for 
telephone interviews. We wanted to know: 

1 . Under what circumstances cooperative learning teams were most successful. The previous 
paper and pencil surveys indicated that, in some places, cooperative learning teams were 
relied on extensively and were seen as being very successful. In other places cooperative 
learning teams were not used at all. We wanted to find out more. 

2. Under what circumstances compendiums of classroom assessments developed by the state 
(called Tool Kits) were successfully used. Again, from the paper and pencil surveys, there 
was a wide disparity between Centers in the emphasis on Tool Kits and perceived usefulness 
of Tool Kits. 

The interviews specifically asked both process and impact questions: 

• The reasons why cooperative learning teams (or Tool Kits) were (or were not) used, barriers 
experienced and how they were addressed, contextual factors the promoted (or inhibited) use, 
and advice they’d give others 

• How the team learning experience (or Tool Kit training) was set up and why it was set up in 
that fashion 

• The impact of the team (or Tool Kit) experience on classroom practice. We asked for 
specific examples 

The interviews also asked questions to corroborate (or not) the information collected in the 
surveys and to gather information that might be related to use (or no use) of teams (or Tool Kits): 

• District contacts’ current understanding of the meaning of quality student assessment and 
confidence in assessing students well. (Interviewing provided a more reliable measure of 
impact than self-report.) 

• Feedback on the state reform effort in general 
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• Other assistance district contacts would like to receive 



We chose the districts to be interviewed based on their responses to groups of questions on the 
surveys. Group one was chosen to represent various combinations of degrees of confidence in 
understanding the characteristics of quality assessment and amount of use of cooperative 
learning teams. Group two was chosen to represent various combinations of degrees of 
confidence in understanding the characteristics of quality assessment and amount of use of Tool 
Kits. 

Site Visits. Site visits were conducted during both the 1996-97 and 1998-99 evaluations. Site 
visits took place in seven districts for the 1996-97 evaluation — some that were moving forward 
comfortably with assessment training and reform and some that were not. The objective was to 
discover the district conditions that encourage productive professional development. Sites were 
nominated by Center staff and selected through other informal communication networks. On- 
site interviews were conducted with district level personnel — superintendent, district assessment 
coordinator, and district curriculum coordinator. Then visits were made to two or three school 
buildings to interview the principal and staff. District staff nominated buildings. Each site visit 
lasted one day. 

For the 1998-99 evaluation, the highest resolution (and least representative) picture was taken 
with site visits. While the previous aspects of the evaluation collected interesting information 
about various groups’ perceptions of the usefulness of the professional development effort, only 
in the site visits was there an attempt to actually gather impact information. This took the form 
of (a) examining actual classroom assessments developed by teachers, and (b) observation in the 
classroom. 

Site visits were conducted in buildings nominated by Centers as being sites in which good things 
were happening and there had been heavy involvement in the professional development effort. 
Thus, the goal was not to document typical ^ impact of professional development on classroom 
assessment practices. Rather the goal was to document what occurred in some of the best sites 
and what they did to get there. 

The two 1998-99 site visits consisted of: 

1 . Interviews with the principals of each building. Questions included the importance of quality 
classroom assessment, what happened and why it happened, barriers and facilitators of 
change, and their knowledge about quality classroom assessment. 

2. Observation in classrooms in each building — observing an assessment process in action, plus 
an interview with each teacher regarding: the assessment observed, the changes in 
assessment going on at their school, barriers and facilitators of change, ability to articulate 
characteristics of quality assessment, and confidence in assessing students well. With 
permission of parents, some students were also interviewed to obtain a notion of the degree 
and type of student involvement in classroom assessment. 

3. Collection of a classroom assessment sample from each teacher in each building willing to 
share — the assessment plus a description of contextual information. These assessments were 
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evaluated for quality by Center staff using a set of rubrics designed for the purpose 4 . Center 
staff members were then invited to draw conclusions about next steps in training. 

Because of the limited funding for the evaluation enterprise in general, results should be 
taken as indicative rather than definitive. 

Results 

The following are a summary of the findings from all sources of evaluation information. 

❖ The program development a large volume of training materials that both Center and District 
Teams found very useful. 

❖ Overall, Center and District Teams were pleased with the services provided to them. 

❖ “Pass-through” training did not work as well as specifying the goals for training and allowing 
Center Teams to development their own workshops and service delivery methods. 

❖ Educators need to see lots of sample assessments aligned with state standards. (This need 
resulted in the development of grade and content specific Tool-Kits .) 

❖ Changes in classroom practice require support from administrators — recognition and support 
for the importance of classroom assessment as well as the WASL, policies that support the 
reform effort, and technology that supports more complex record keeping. It is not enough to 
just include professional development of teachers. 

❖ Building a sound foundation of basic assessment knowledge made understanding the 
EALRs, the WASL, and other assessment materials developed through the project team 
more likely. Those who studied the WASL, EALRs and other materials without this 
understanding were not as able to use these things well. 

❖ The most important determinant of how much educators learned about assessment was the 
depth with which they studied the topic — those who were in cooperative learning teams 
learned the most. 

❖ The state-assessment topics (EALRs and WASL), seeming more immediate and of higher 
stakes, tended to crowd out the more long term assessment literacy training needed to 
underpin a standards-based educational environment. 

Recommendations 

As the result of the accumulation of lessons learned from both formal and informal information 
sources, major recommendations were made in the project final report (May 2000) concerning 
guiding philosophy, professional development strategies to continue and discontinue, content to 
continue and discontinue, and potentially important new efforts. These recommendations 
included: 



4 



These rubrics can be obtained from the sources cited earlier. 
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1 . Sometimes the need to pass on state assessment (WASL) information quickly rendered basic 
assessment literacy training (understanding the characteristics of high quality assessment and 
how to use it to help students learn) a lower priority. However, as evaluation results 
indicated, a basic understanding of high quality assessment was needed by all educators in 
order to respond most productively to the other aspects of the state reform effort. This 
“Catch-22” resulted in the recommendation that basic assessment literacy training be 
separated from WASL information flow. Specifically, the state should discontinue 
disseminating information about the state assessment through the Centers. Direct 
communication with districts and buildings is more effective. 

2. A balance of emphasis on large-scale and classroom assessment is essential if educators are 
to respond most productively to the state reform effort. The WASL information flow 
overpowered attention to ensuring that teachers have the assessment skills they need to 
develop the high quality, student-involved classroom assessment practices necessary in a 
standards-based educational environment. Again, this implied the need to separate WASL 
and classroom assessment training. 

3. Local leadership is essential to develop productive assessment environments that maximize 
school effectiveness/student achievement. Superintendents and principals need to understand 
the need for balance between large scale and classroom assessment and the need for 
professional development in assessment. They need to have a vision of what it looks like 
when assessment systems are balanced and how to create it. So, hand-in-hand with 
assessment literacy training needs to be assistance with developing comprehensive local 
assessment systems. 

4. Continue producing Tool Kits of sample assessments for use in the classroom. But, 
remember that these Tool Kits have less value if teachers don’t come to them with a 
foundational understanding of high quality, student-involved classroom assessment. Without 
this understanding, teachers tend to use sample assessments in a rote fashion without the 
ability to generalize. 

5. Continue to build partnerships between professional organizations, colleges and universities, 
the Centers, and others providing professional development in assessment. 

6. Be careful with “pass-through” training in which fixed workshops are progressively “taught” 
to trainers in a pyramid fashion from the original developers to district training teams. It 
works better to specify the goals of professional development and allow Centers to use 
provided materials and their own expertise to design training tailored to their own styles and 
clientele. 

7. Learning about assessment best takes place in the context of learning teams. Centers should 
focus on this professional development option. 



Washington Professional Development Program in Assessment Literacy, NCME, Seattle, 2001 
Judy Arter, Assessment Training Institute, 503-228-3060 



7 



TM033181 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 

Educational Resources Information Center (ERIC) 

REPRODUCTION RELEASE 




(Specific Document) 



I. DOCUMENT IDENTIFICATION: 



Title: 



: /Tsvess » 

QslSuJ-{s 

Author(s): 



Corporate Source: 



i Source: j . , . 

tfTX, 5 T> 500 2^, Scci-Zc 3 00; 



Publication Date: 

Ajn nl y loo/ 



II. REPRODUCTION RELEASE: 

In order to disseminate as widely as possible timely and significant materials of interest to the educational community, documents announced 
in the monthly abstract journal of the ERIC system Resources in Education ( RIE), are usually made available to users in microfiche, reproduced 
paper copy, and electronic media, and sold through the ERIC DocumentReproduction Service (EDRS). Credit is given to the source of each 
document, and, if reproduction release is granted, one of the following notices is affixed to the document. 

If permission is granted to reproduce and disseminate the identified document, please CHECK ONE othe following three options and sign 
at the bottom of the page. 

The sample sticker shown below will be The sample sticker shown below will be 

affixed to all Level 1 documents affixed to all Level 2A documents 

PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE, AND IN ELECTRONIC MEDIA 
FOR ERIC COLLECTION SUBSCRIBERS ONLY, 

HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

2A 

Level 1 Level 2A 

O' □ 

Check here for Level 1 release, permitting Check here for Level 2A release, permitting reproduction Check here for Level 2B release, permitting 

reproduction and dissemination in microfiche or other and dissemination in microfiche and in electronic media reproduction and dissemination in microfiche only 

ERIC archival media (e.g., electronic)anc/ paper copy. for ERIC archival collection subscribers only 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 





TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

1 



The sample sticker shown below will be 
affixed to all Level 2B documents 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE ONLY HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



2B 



Level 2B 



□ 



Documents will be processed as indicated provided reproduction quality permits. 

If permission to reproduce is granted, but no box is checked, documents will be processed at Level 1 . 



Sign 

here? 

please 



1 hereby grant to the Educational Resources Information Center (ERIC) nonexclusive permission to reproduce and disseminate thi 
document as indicated above. Reproduction from the ERIC microfiche or electronic media by persons other than ERIC employees an 
its system contractors requires permission from the copyright holder. Exception is made for non-profit reproduction by libraries anc 
other service agencies to satisfy information needs of educators in response to discrete inquiries. 


Signature: • fl /V t 


Printed Name/Position/Title: 

■JtxdTf'Tf A.Avfe*. 


Organ iz£tio\/Ad dressy r 

( ) S&e cUatrve- 





O 

ERIC 




E-^1ail Address: 

j 



(s>[( *(b I 



0.&&ASS as*!, cervix 



III. DOCUMENT AVAILABILITY INFORMATION (FROM NON-ERIC SOURCE): 

ff permission to reproduce is not granted to ERIC, or, if you wish ERIC to cite the availability of the document from another source, please 
provide the following information regarding the availability of the document. (ERIC will not announce a document unless it is publicly available, 
and a dependable source can be specified. Contributors should also be aware that ERIC selection criteria are significantly more stringent for 
documents that cannot be made available through EDRS.) 



Publisher/Distributor: 



Address: 



Price: 



IV. REFERRAL OF ERIC TO COPYRIGHT/REPRODUCTION RIGHTS HOLDER: 

If the right to grant this reproduction release is held by someone other than the addressee, please provide the appropriate name and address: 




V. WHERE TO SEND THIS FORM: 



Send this form to the following ERIC Clearinghouse: 



However, if solicited by the ERIC Facility, or if making an unsolicited contribution to ERIC, return this form (and the document being contributed) 
to: 



EFF-088 (Rev. 2/2000) 

o 

ERLC 



ERIC Processing and Reference Facility 
4483-A Forbes Boulevard 
Lanham, Maryland 20706 

Telephone: 301-552-4200 
Toll Free: 800-799-3742 
FAX: 301-552-4700 
e-mail: ericfac@inet.ed.gov 
WWW: http://ericfac.piccard.csc.com 



